Graphics and Cognition

How Do We Perceive Charts?

Susan Vanderplas

2023-10-12

Outline

  1. A quick cognitive psychology primer

  2. Applications to Charts

  3. Models of graph cognition

Cognitive Psychology Primer

A sketch from Descartes showing a stimulus, eyes, optic nerve, and higher level brain regions artistically. External information is contained in a box labeled 'Stimulus' on the right side of the image. The eyes and optic nerves are contained in a box labeled 'Perception' in the middle. Finally, the higher-level brain regions are contained in a box labeled 'Cognition'. An arrow at the bottom of the image labeled Time moves from right to left.

Attention

  • Focused vs. Divided attention

    • Divided = multiple input streams must be processed (e.g. driving a car)

    • Focused = single input stream

  • Attention shifts with gaze systematically throughout a visual stimulus

  • Information which is extraneous to the task may not be stored or remembered

Written text. An annotation layer on top shows how the eye fixates sequentially on each word/syllable (approximately) and then jumps to the next word in the sequence.

Introspection

As you look at the chart I’m about to show you, what do you focus on first? second? third?

Mentally record how you read in the chart and what you’re looking at over time.

Most of the time, we are concerned with attentive perception. Some factors are processed preattentively (without conscious attention), but for everything else, introspection is a valid observational method.

Introspection

Memory

Long Term Memory

Long Term Memory Information Retrieval

  • unlimited storage capacity (theoretically)
  • maintained for long periods
  • two types:
    • declarative/explicit - you can consciously evoke this
    • non-declarative/implicit - procedural memories, such as ‘how to ride a bike’

Memory

Short Term Memory

  • Not a term used any more. Replaced by “Working Memory”

Memory

Working Memory

Image from https://www.simplypsychology.org/working-memory.html

  • Proposed by Baddeley and Hitch (1974)
  • Limited capacity
  • Components are thought to be relatively independent of each other

Application to Charts

Perception

  • Sensory input
  • Recognition of type of chart
  • Takes place very quickly - preattentively

A scatterplot with points, an x-axis label, a y-axis label, and a title that says 'Title with context'. The word perception is added on top of the scatterplot.

Gestalt Grouping

  • Gestalt heuristics used to group points into clusters/trends

A scatterplot with points, an x-axis label, a y-axis label, and a title that says 'Title with context'. There is a main set of points surrounded by a gray region that contains the points, indicating that these points are part of a group. There is a second group of 3 points surrounded by an orange dot that appear to be completely separate from the main group.

Integration with Domain Knowledge

  • Begin to assign meaning to the relationship between the data and the chart labels/titles

  • Retrieve knowledge from long term memory and make that available in working memory

A scatterplot with points, an x-axis label, a y-axis label, and a title that says 'Title with context'. The titles are highlighted and arrows are drawn between them. A text label of 'integration with domain knowledge' is shown between the arrows and titles overtop the data.

Integration with Domain Knowledge

Domain knowledge includes:

  • How graphs are usually structured (e.g. y-axis increases from bottom to top)

  • Conventions for use of filled vs. empty space

  • Knowledge of relationships, events, etc. that might be impactful

Figure from Padilla et al. Cognitive Research: Principles and Implications (2018) 3:29

Storytelling

  • Assign meaning to relationships

  • Fit visual statistics

  • Look for things that do and don’t fit a rough working hypothesis about the data

A scatterplot showing a positive linear association between X and Y. The points are surrounded by a bounding region and a dotted regression line is drawn through the center of the region, with solid lines at the edges of the region, indicating a rough regression fit. A group of 3 points at the bottom right of the plot is colored blue and is labeled 'outlier assessment'.

Inference

  • Numerical estimation from the chart with uncertainty

  • Visual search, followed by estimation, calculation, and inference.

A scatterplot showing a positive linear association between X and Y. A dotted line indicates a location on the x-axis, 'x'. A red band around the line is labeled visual search, and an arrow points to the dotted line, which is labeled estimation. A yellow line points to the corresponding points around the dotted line, labeled 'calculation', and a purple line labeled 'inference' points to a region on the y-axis that has a corresponding confidence interval.

Prediction and Application

  • Moving beyond the data shown to interpret and apply meaning

  • Draw conclusions

  • Make predictions about the future

A scatterplot showing a positive linear association between X and Y. A regression line shows the approximate mean of the points shown, with a dotted extension of the line which is labeled prediction. An orange arrow points from the prediction segment to an image of the globe, which is labeled application.

Decision Making with Graphics

What is the average number of Adelie and Chinstrap penguins measured?

What steps do you go through to calculate this average?

General Process

From Padilla et al (2018), an adaptation of Pinker (1990)

Adding in Working Memory

From Padilla et al (2018)

Fast and Slow Decisions

From Padilla et al (2018)

Fast Decisions

Visual heuristics replace calculations, requiring very little working memory

Slow Decisions

Working memory required for each estimation and calculation step

Cross-domain Findings

Major takeaways from different experiments across domains and application areas:

  1. Visualizations direct viewers bottom-up attention, which can both help and hinder decision making

  2. The visual encoding technique gives rise to visuospatial biases

  3. Visualizations that have greater cognitive fit produce faster and more effective decisions

  4. Knowledge-driven processes can interact with the effects of the encoding technique.

Directing bottom-up attention

From Padilla et al. (2018)

Directing bottom-up attention

Attention focused on the pictogram instead of the base rate, leading to decisions that are suboptimal.

From Padilla et al. (2018)

Directing bottom-up attention

If a path intersects with a point of interest, resulting decisions tend to be biased.

From Padilla et al. (2018)

Visual-spatial Biases

From Padilla et al. (2018), rearranged to fit on slides

Biases:

  • Anchoring

  • Anecdotal evidence

  • Containment - what is in the container is different from what is outside the container. Ex: Binning continuous data

  • Deterministic construal - what is shown is deterministic instead of probabilistic

  • Quality bias - high quality image = high quality science

Cognitive Fit

If there is a mismatch between the visualization type and the decision making component, working memory must be used to compensate

From Padilla et al. (2018), reproduced from Hegarty et al. (2012)

Knowledge-Driven Processing

  • Instructions/training (short term knowledge)

  • Individual differences (e.g. math skills, background knowledge, interests)

  • Knowledge can be used to overcome e.g. familiarity bias (preference for familiar but ineffective visualizations)

  • Subject-matter expertise

From Padilla et al. (2018), a reproduction of stimuli used by Galesic et al (2009).